Set the inference gateway Service to externalTrafficPolicy Cluster#320
Merged
negz merged 1 commit intoJul 7, 2026
Merged
Conversation
Envoy Gateway defaults the managed LoadBalancer Service to externalTrafficPolicy: Local. Some clouds' load balancers reject that: on Nebius the Service emits SyncLoadBalancerFailed and never gets an external IP, so the gateway address stays pending, the ServingStack and InferenceCluster never reach Ready, and deployments targeting the cluster sit unscheduled. It was previously worked around by manually patching the Service. Compose an EnvoyProxy pinning envoyService.externalTrafficPolicy to Cluster and reference it from the GatewayClass via parametersRef, the idiomatic Envoy Gateway way to configure the managed Service. Mark the EnvoyProxy Object ready alongside the other gateway resources so it does not hold the composite from Ready. Cluster is accepted by every cloud the provider runs on (GKE, EKS, Nebius) and the inference gateway does not need client source-IP preservation. Updates the golden tests. Fixes modelplaneai#316. Signed-off-by: Pablo Luna <pablo@upbound.io>
negz
approved these changes
Jul 7, 2026
Collaborator
|
Thanks @pluna this looks good to me. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of your changes
Envoy Gateway defaults the managed LoadBalancer Service to
externalTrafficPolicy: Local. Some clouds' load balancers reject that: on Nebius the Service emitsSyncLoadBalancerFailedand never gets an external IP, so the gateway address stays pending, the ServingStack and InferenceCluster never reach Ready, and deployments targeting the cluster sit unscheduled. It was previously worked around by manually patching the Service after the fact.This composes an
EnvoyProxythat pinsprovider.kubernetes.envoyService.externalTrafficPolicytoClusterand references it from theGatewayClassviaparametersRef, the idiomatic Envoy Gateway way to configure the managed Service.Clusteris accepted by every cloud the provider runs on (GKE, EKS, Nebius), and the inference gateway does not need client source-IP preservation. The composedgateway-proxyresource is marked ready inmark_readinessalongside the other gateway resources so it does not hold the composite from Ready.The change is composition-only with no cloud branching, so it behaves identically across clouds. I validated it live end to end on EKS (an L4
g6.xlargecluster): the managed gateway Service came up withexternalTrafficPolicy: Clusterand a real ELB address, the InferenceCluster reached Ready, and Qwen3-8B served completions through the gateway. The golden tests cover the composed EnvoyProxy, the GatewayClassparametersRef, and the readiness transition.Fixes #316.
I have:
nix flake check(or./nix.sh flake check) and made sure it passes.git commit -s.